Introduce -Zsplit-metadata option #120855

bjorn3 · 2024-02-09T18:48:03Z

This will split the crate metadata out of library files. Instead only the svh and a bit of extra metadata is preserved to allow for loading the right rmeta file. This significantly reduces library size. In addition it allows for cheaper checks if different library files are the same crate.

A fair amount of the complexity in this PR is to work around the fact that cargo doesn't directly support this option yet.

Fixes #23366
Closes #29511
Fixes #57076

Revives #93945

rustbot · 2024-02-09T18:48:11Z

r? @cjgillot

rustbot has assigned @cjgillot.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

bjorn3 · 2024-02-09T18:49:19Z

compiler/rustc_metadata/src/locator.rs

                    rlibs.insert(loc_canon, PathKind::ExternFlag);
                } else if loc.file_name().unwrap().to_str().unwrap().ends_with(".rmeta") {
                    rmetas.insert(loc_canon, PathKind::ExternFlag);
                } else {
+                    rmetas.insert(loc_canon.with_extension("rmeta"), PathKind::ExternFlag);


This is a hack. It should probably be replaced with proper lookup in the library search path, or with cargo passing --extern twice for both the rlib/dylib and the rmeta file.

bjorn3 · 2024-02-09T18:51:31Z

src/bootstrap/src/core/build_steps/compile.rs

+            if extension == "so" || extension == "dylib" {
+                // FIXME workaround for the fact that cargo doesn't understand `-Zsplit-metadata`
+                toplevel.push((file_stem.clone(), "rmeta".to_owned(), None));
+            }


This together with the rustc wrapper passing --emit metadata is a big hack working around the fact that cargo doesn't support -Zsplit-metadata and as such doesn't cause a .rmeta file to be emitted and also doesn't track it in the compiler-artifact json message. This also causes issues with recompilations as the loop below may copy outdated rmeta files into the sysroot, which would then result in compiler errors.

bjorn3 · 2024-02-09T18:52:18Z

tests/ui/duplicate_entry_error.rs

@@ -1,4 +1,4 @@
-// normalize-stderr-test "loaded from .*libstd-.*.rlib" -> "loaded from SYSROOT/libstd-*.rlib"
+// normalize-stderr-test "loaded from .*libstd-.*.rmeta" -> "loaded from SYSROOT/libstd-*.rmeta"


The .rmeta file is now considered the canonical one as the .rlib doesn't contain any crate metadata beyond the header.

bjorn3 · 2024-02-09T18:52:35Z

@bors try @rust-timer queue

bors · 2024-02-09T18:53:44Z

⌛ Trying commit 7770002 with merge 8fb65ac...

Introduce -Zsplit-metadata option This will split the crate metadata out of library files. Instead only the svh and a bit of extra metadata is preserved to allow for loading the right rmeta file. This significantly reduces library size. In addition it allows for cheaper checks if different library files are the same crate. A fair amount of the complexity in this PR is to work around the fact that cargo doesn't directly support this option yet. Fixes rust-lang#23366 Fixes rust-lang#57076 Revives rust-lang#93945

bjorn3 · 2024-02-09T18:54:55Z

What would be the best way to handle the necessary cargo changes? We can't use them until the bootstrap compiler includes the cargo changes, but cargo would depend on -Zsplit-metadata which doesn't exist yet. Would unconditionally creating a .rmeta file for dylibs like we do for rlibs be fine? Or should I add a -Zsplit-metadata option to cargo without it getting tested and then once the bootstrap bump happens, change this PR to use this option? Or land -Zsplit-metadata first without actually making any use of it, then update cargo and finally make use of it in the standard library?

bors · 2024-02-09T20:22:41Z

☀️ Try build successful - checks-actions
Build commit: 8fb65ac (8fb65acf1a11f547302c62c93bf83cca3ff4cd9d)

rust-timer · 2024-02-09T23:49:26Z

Finished benchmarking commit (8fb65ac): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	1.4%	[0.3%, 2.7%]	7
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.4%	[-0.5%, -0.4%]	3
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	7.1%	[5.4%, 8.4%]	4
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-5.2%	[-5.2%, -5.2%]	1
All ❌✅ (primary)	-	-	0

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 666.509s -> 669.311s (0.42%)
Artifact size: 307.99 MiB -> 301.34 MiB (-2.16%)

bjorn3 · 2024-02-10T14:17:08Z

This should save about 4MB on the download size of a toolchain without any extra targets other than the host:

component	`e28fae5` (no split metadata)	`8fb65ac` (split metadata)	diff
rustc	70.54MB	68.39MB	-2.15MB
rust-std	27.63MB	25.61MB	-2MB
rustc-dev	110.84MB	110.90MB	+0.06MB

For Bevy the size of the target dir shrinks a lot:

mode	normal build	`--emit metadata -Zsplit-metadata`	diff	relative diff
debug mode	4.6GiB	4.3GiB	-300MiB	-6.5%
debug mode (no incr comp)	2.8GiB	2.6GiB	-200MiB	-7%
release mode	843.6MiB	614.3MiB	-230MiB	-27%

bors · 2024-02-16T09:54:33Z

☔ The latest upstream changes (presumably #120486) made this pull request unmergeable. Please resolve the merge conflicts.

petrochenkov · 2024-03-07T15:10:42Z

@bjorn3
Could you

Split 2eb24d9 into a separate PR in case it causes regressions
Split all preparatory refactorings like 1065559 into a separate PR
Rebase the remaining changes, make CI green, cleanup history, and add FIXME comments to logic that only exists due to lacking cargo support

(I'll be able to review the first two PRs quickly.)
@rustbot author

bors · 2024-10-18T09:06:15Z

☔ The latest upstream changes (presumably #131869) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2024-12-27T15:58:48Z

☔ The latest upstream changes (presumably #134822) made this pull request unmergeable. Please resolve the merge conflicts.

This will split the crate metadata out of library files. Instead only the svh is preserved to allow for loading the right rmeta file. This significicantly reduces library size. In addition it allows for cheaper checks if different library files are the same crate.

petrochenkov · 2025-02-19T15:39:53Z

Status is not very clear, still waiting on author.
@bjorn3 Do you plan to work on completing it this year?

bjorn3 · 2025-02-19T15:43:32Z

I do still plan on working on this.

petrochenkov · 2025-03-13T13:23:56Z

Closing in favor of #137535.

Introduce `-Zembed-metadata` to allow omitting full metadata from rlibs and dylibs This is a continuation of rust-lang#120855 (I was mentored by `@bjorn3` to move it forward). Most of the original code was written by bjorn3, I tried to clean it up a bit and add some documentation and tests. This PR introduces a new unstable compiler flag called `-Zembed-metadata=[no|yes]`, with the default being `yes` (see rust-lang#57076 for context). When set to `no`, rustc will only store a small metadata stub inside rlibs/dylibs instead of the full metadata, to keep their size smaller. It should be used in combination with `--emit=metadata`, so that the users of such a compiled library can still read the metadata from the corresponding `.rmeta` file. [This comment](rust-lang#120855 (comment)) shows an example of binary/artifact size wins that can be achieved using this approach. Contrary to rust-lang#120855, this PR only introduces the new flag, along with a couple of run-make tests and documentation, but does not yet use it in bootstrap to actually compile rustc. I plan to do that as a follow-up step (along with integration in Cargo, which should ideally just always pass this flag to reduce the size of target directories). Fixes rust-lang#23366 Closes rust-lang#29511 Fixes rust-lang#57076 Another attempt of rust-lang#93945 and rust-lang#120855. r? `@petrochenkov`

rustbot assigned cjgillot Feb 9, 2024

bjorn3 commented Feb 9, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Feb 9, 2024

bjorn3 mentioned this pull request Feb 9, 2024

Introduce -Zsplit-metadata option #93945

Closed

This comment has been minimized.

Sign in to view

petrochenkov self-assigned this Feb 9, 2024

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Feb 9, 2024

cjgillot removed their assignment Feb 18, 2024

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 7, 2024

bjorn3 mentioned this pull request Mar 8, 2024

Remove a workaround for a bug #122186

Closed

bjorn3 force-pushed the split_metadata4 branch from a5c9a9c to f912043 Compare November 1, 2024 11:31

This comment has been minimized.

Sign in to view

bjorn3 force-pushed the split_metadata4 branch from f912043 to d093c9d Compare November 1, 2024 12:35

This comment has been minimized.

Sign in to view

bjorn3 force-pushed the split_metadata4 branch from d093c9d to fe906a0 Compare November 27, 2024 15:25

This comment has been minimized.

Sign in to view

bjorn3 force-pushed the split_metadata4 branch from fe906a0 to d918933 Compare January 9, 2025 13:42

This comment has been minimized.

Sign in to view

bjorn3 force-pushed the split_metadata4 branch from d918933 to 3d17fbe Compare January 14, 2025 15:23

This comment has been minimized.

Sign in to view

Simplify find_commandline_library

8656e1b

bjorn3 force-pushed the split_metadata4 branch 2 times, most recently from 0e3b1b5 to 32bd529 Compare January 14, 2025 16:13

bjorn3 and others added 3 commits January 14, 2025 16:15

Fix stdlib building

50c1208

Rustbuild: enable -Zsplit-metadata for stage != 0

bedeb19

bjorn3 force-pushed the split_metadata4 branch from 32bd529 to bedeb19 Compare January 14, 2025 16:17

This comment has been minimized.

Sign in to view

Fix cg_clif tests

6720404

Kobzol mentioned this pull request Feb 24, 2025

Introduce -Zembed-metadata to allow omitting full metadata from rlibs and dylibs #137535

Merged

petrochenkov closed this Mar 13, 2025

bjorn3 deleted the split_metadata4 branch March 13, 2025 13:32

Kobzol mentioned this pull request Mar 14, 2025

Add -Zsplit-metadata flag to reduce disk usage rust-lang/compiler-team#851

Closed

3 tasks

Kobzol mentioned this pull request Apr 2, 2025

Add support for -Zembed-metadata rust-lang/cargo#15378

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce -Zsplit-metadata option #120855

Introduce -Zsplit-metadata option #120855

bjorn3 commented Feb 9, 2024 •

edited

Loading

rustbot commented Feb 9, 2024

bjorn3 Feb 9, 2024

bjorn3 Feb 9, 2024

bjorn3 Feb 9, 2024

bjorn3 commented Feb 9, 2024

This comment has been minimized.

bors commented Feb 9, 2024

bjorn3 commented Feb 9, 2024 •

edited

Loading

This comment has been minimized.

bors commented Feb 9, 2024

This comment has been minimized.

This comment has been minimized.

rust-timer commented Feb 9, 2024

bjorn3 commented Feb 10, 2024

bors commented Feb 16, 2024

petrochenkov commented Mar 7, 2024 •

edited

Loading

bors commented Oct 18, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bors commented Dec 27, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

petrochenkov commented Feb 19, 2025

bjorn3 commented Feb 19, 2025

petrochenkov commented Mar 13, 2025

		@@ -1,4 +1,4 @@
		// normalize-stderr-test "loaded from .libstd-..rlib" -> "loaded from SYSROOT/libstd-*.rlib"
		// normalize-stderr-test "loaded from .libstd-..rmeta" -> "loaded from SYSROOT/libstd-*.rmeta"

Introduce -Zsplit-metadata option #120855

Introduce -Zsplit-metadata option #120855

Conversation

bjorn3 commented Feb 9, 2024 • edited Loading

rustbot commented Feb 9, 2024

bjorn3 Feb 9, 2024

Choose a reason for hiding this comment

bjorn3 Feb 9, 2024

Choose a reason for hiding this comment

bjorn3 Feb 9, 2024

Choose a reason for hiding this comment

bjorn3 commented Feb 9, 2024

This comment has been minimized.

bors commented Feb 9, 2024

bjorn3 commented Feb 9, 2024 • edited Loading

This comment has been minimized.

bors commented Feb 9, 2024

This comment has been minimized.

This comment has been minimized.

rust-timer commented Feb 9, 2024

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Instruction count

Max RSS (memory usage)

Cycles

Binary size

bjorn3 commented Feb 10, 2024

bors commented Feb 16, 2024

petrochenkov commented Mar 7, 2024 • edited Loading

bors commented Oct 18, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bors commented Dec 27, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

petrochenkov commented Feb 19, 2025

bjorn3 commented Feb 19, 2025

petrochenkov commented Mar 13, 2025

bjorn3 commented Feb 9, 2024 •

edited

Loading

bjorn3 commented Feb 9, 2024 •

edited

Loading

petrochenkov commented Mar 7, 2024 •

edited

Loading